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Framework for the Educational Evaluation of 
Computer**As3isted Instruction 
Donald L. Aidennan 
Educational Testing Service 

Introduction 

The PLATO and TICCIT demonstrations will be analyzed with a view to 
determining their educational effectiveness. The following report presents 
an overview of how this will be accomplished, in full awareness that the 
concept of "effectiveness" has been notoriously subject to diverse inter- 
pretation. To avoid misunderstandings, we are giving the concept as much 
prior elaboration and specification as is now possible. As we see it, the 
entire context of our eyaluation of educational effectiveness will be 
clarified by the delineation of four essential issues, which for the sake of 
brevity we pose as four questions that must be answered: the vh at , who , how , 
and when of the evaluation. Together these basic questions form the dimensions 
of our analytical approach to evaluating the educational effectiveness of PLATO 
and TiCCIT. 

We first identify what areas of inquiry are appropriate to an evaluation of 
computer-based education. The cognitive and affective domains of student perform- 
ance and faculty acceptance represent a gross classification of this "what" 
dimension. For thfe cognitive domain, our primary concern will be how the students 
achievement and behavior have been affected. For the affective domain, we shall 
seek important information on the effects of an innovative technology through 
analyzing reaction to computer curricular materials and to the medium itself. 
The constituents of both domains will be successively refined and ordered in 
a hierarchical manner. This greater precision and detail will yield indications 



of wliat subtopics and item-stems are relevant to effectiveness. Breadth of 
coverage will hopefully ensure our ol?jectivity; detail will permit a telling 
probe of the shibboleths of computer-assisted instruction as reflected by 
PLATO and TICCIT. 

The se^^eral audiences concerned with the demonstrations will constitute 

the **who** dimension of the educational analysis* Because of their differing 

perspectives and preconceptions^ each audience will serve as an important . 

source of data* Depending on the area of inquiry, the sources of critl'i^l 

information will be the educational ins t i tut ion > government, industry, and 

the public* This probe of different societal sectors will lead not only to 

the documentation of diverse viewpoints on educational ef fectiv-tness., but 
ft 

also to supportive information for the cost: and technical analyses. The expres- 
sion of affective and cognitive impact across vested interests and characteristics 
is fundamental to the concept of evaluation; this enables us to study attitudes 
toward coi^uter*>assisted instruction and educational practices across a range 
of viewpoints (e*g*, students, instructors, and administrators) » and to examine 
student performance across a range of ability levels and traits. The ,C4ner 
sifting of tUj dimension will permit us to focus on what comparison groups will 
be needed and on what grpups concerned with the demonstrations must be represented* 

The systematic administration of instruments in each area of inquiry and 
to^-veach appropriate audience requires a plan of how it is to be in^lemented. 
How instruments are to be administered is of course determined by available 
modes of data acquisition. At present these include: tests, student records, . 
questionnaires, online systems, interviews, and observations. Each mode carries 
particular considerations for instrumentation, ranging from classical test 
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attributes (reliability and validity) to personnel and financial demands. 
Whether a mode is appropriate for the collection of cognitive and affective 
data wi:.l depend on what detail of analysis is desired for a particular audience. 

Precisely when an instrument should be administered is, similarly, an 
Important consideration. Since baseline control depends on measures' takiH 
"belore the introduction of computer-^asrsisted instruction and concurrent 
control on measures taken during the actual demonstrations » both (>rocedures 
rely heavily on coordinating measurements with the timing of administrations* 
Data collection at specified time intervals is also critical td ol taining trend 
information. We reali£e the "when** dimension Is a particularly sensitive one* 
We would not want, for example, to fdrce premature judgments or to measure 

achievement prematurely » nor would we wish» to burden respondents with excessive 

'I '* ^ 

demands upon their time and attention* 

Certainly there are interrelationships among the preceding dimensions (i.e., 
areas of inquiry » audiences^ modes of data acquisition, and £ime of data acquisi-^ 
tion) of the educational analysis, as indicated in Figure !• Although many 

A, 

combinations of components along these dimensions .would be inappropriate for 
our purposes > the individual cell.»3 in this initial schematic representation of 
what, who, and how begin to identify information critical for assessing effective- 
ness. The accumulation of that information will depend, however, upon our 
first identifying (1) the area of inquiry; (2) the responding audience; (3) 
the instrument form used for data accuisition; and (4) the time of administering 
the instrument. Through such specification, data can be gathered that attest 
to tlie educational strengths and weaknesses of the demonstrations* 

Insert Figure 1 about here 



Dimensions 

A» Areas of Inquiry 

To capture what impact the demonstrations have had upon cognitive skills 

ft 

and attitudes, we must isolate indicators from the cognitive and affective 
doiaains. For the cognit.ve domain indications of instructional effectiveness 
will be ^athejred through various achievement measures » including standardized 
te;3t8» objective-based tests, and measures of course performance. We shall 
.also supplement information on student achievement with behavioral data on 
lesson compl^tfion, instructional sequence, and other descriptions of cognitive 
approach toward. the subject matter.. Activities of students and instructors 
wlU serve as indicators of indirect effects beyond instructional outcomes, 
such as changes in the allocation of effort and time. Further specification 
on achievement measures will be possible after the elaboration of course 
objectives. 

We shall substantiate and extend information from the cognitive domain 
through analysis of empirical data collected in the affective domain. The 
response data relevant to our assessment of educational impact are subsumed 
under the categories of courseware, role, and appraisal of computer-based 
education. Courseware refers to the instructional material employed in the 
demonstrations; the category encompasses content from the subject area, 
instructional strategy, mode of delivery to the student, anci procedures for 
producing instructional materials. Role refers to the manner in which computer- 
assisted instruction is used in the school, with special attention to attitudes 
and receptivity toward that use. Appraisal concerns the prioyicies and basis 
for evaluation held by different audiences. These three classifications of 
inquiry are intended as guidelines in the development of ^i terns for the instruments* 
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The delineation of the areas of inquiry from broad concepts to specific 
indicators is illustrated in Figure 2. As may be seen, ainbiguous .and sometimes 
elusive concepts are brought through successive refinements, to the level of 
item-stems. Priorities in the evaluation are implied by lef t-to-rlght 
positioning a^ong each level of the hierarchy. Since courseware influences 
cognitive effects and is amenable to review by subject-matter experts and 
instructional psychologists, a relationship between the cognitive domain and 
courseware is depicted. Brackets enclose "production" to emphasize that 
responses in this area pertain only to logistical, not instructional, issues 
in the development of computer curricular materials. The hierarchical schema 
presents a partial elaboration of the "what" dimension in the educational 
analysis. Though incomplete, the portrayed hierarchy is intended to convey 
both the depth and breadth of inquiry; it also begins to specify priorities 
for the allocation of our resources. The data analyses will serve to high- 
light certain information, from our broad coverage, which reveals the strengths 
and weaknesses of the respective denK>ns trations • 

Insert Figure 2 about here 

b> Audiences 

Within the context of planning the educational analysis, audiences (see 
Figure 3) are meant to denote those persons from whom we shall gather responses 
in the various areas of inquiry. To allow the expression of significant view- 
points we shall include respondents who are representatives of the educational 
institution, government, public, and industry. Potential data sources in 
these sectors are the following: (i) students, instructors, and administrators 
in the cooperating schools and colleges; (2) state and local boards of education 
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responsible for policies in those schools and colleges; (3) parents of children 
in the site schools and visltots to the demonstration sites; (4) committees 
that advise community colleges on the content areas; and (5) manufacturers of 
system components for PLATO and TICCIT. The foci . our attention during the 
baseline period ip. the educational institution, i.e., the cooperating schools 
and col];eges. 

Certain characteristics of students, instructors and administrators are' 

relevant to our identifying appropriate audiences and to our establishing 

proper comparison groups. The first such characteristic is participant 

status in the demonstrations. For s,tudents, participation is determined by 

enrollment in classes or courses scheduled to use C/LI programs; for instructors, 

by responsibility for the use of CAI in a course or class; for administrators, 

by implementation of CAI under their auspices. The additional consideration of 

extent of involvement according to participant criteria depends upon varla^iions 

/ 

within audiences among participants. For those students and instructors who 
have no direct contacts with CAI another attribute is the similarity between 
computer-based and conventional instruction. This similarity of courses 
entails an examination of overlay in instructional objectives anjJ resource 
materials. Identification of students in, and instructors of courses which 
parallel computer-assisted instruction permits us to compare instructional 
outcomes. 

Further classifications of respondents within the educational institution 
relate to the particular district, school, and course or department^ This 
information is require^d to investigate effectiveness across content areas, 
schools, and distrii:trS, and to recognize natural differences among audiences. 

ERIC 
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Although a breakdown by districts, schools, and courses or departments appears 
appilcafciu to students and instructors^ an administrative position usually 
satisfies only one of these categories accarding to the duties of that 
adminiscratlve office. 

Insert Figure i about here 

uniiie the above delineation of the "who" dimension suffices for a 
preliminary identif ication» other characteristics are certainly important in 
establishing comparable or matched groups for, analysis . Beyond descriptive 
inforoation for classification, traits such as student aptitude or instructor * 
experience might influence achievement or at;titude. Random assignment, matched 
groups, and covariance offer means to overcome anticipated control difficulties 
which weans are employed remains to be arranged although random assignment is 
certainly the preferred procedure. The available control procedures and the 
multi'iude of potential respondents suggest that comparison groups for computer- 
basea and co-nventlonai instruction are accessible. li\ addition,^'' tut use of 
baseiine and concurrent comparisons for audience classifications provides a 
view of difference and change attributable to computer-assisted instruction. 

<■ ' 

' '^^^^^^ Uata ALquisilion 
lue modes tor a^iulring data for the educational analysis are enumerated 
as: tests, records, questionnaires, online systems, interviews and obser- 
vations. Tests in standardized and objective-based form are appropriate for 
the instrumentation of cognitive inquiries* This information on student 
achievement and, possibly, aptitude will be complemented by records and online 
svfiLems* Scuooi records might include additional data on achievement and 
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aptituda by providing course grades and test scores; fhey i&ay furnish supple- 
iDentffl information on approach by supplying data on school attendance. If we 
have access to the files and if we find that' comprehensiW testing programs 
alread> exist, the; necessity to administer standardized tests will be eliminated. 
Online systems provide pertinent achievement and' behavior information through 
various evaluations of student progress, and descriptions of student inter- 
action and sequence through courseware. The abundance of bitine information 
encompasses: performance on reviews, exercises, and tests; <:om^letion of 
required and optional materials; latenciesi time spent at terminal. Systematic 
observations, especially in the elementary scliools, allow the accumulation of 
behiv-^oral data. 

Questionnaires and interviews will constitute the means for ^ta oollection 
in the affective domain. The standard summative or Likert scale is a suitable 
format for attitude items in-^'questionnaircs. A survey instruroeot in a multiple- 
choice format is appropriate ^or gathering Information on audience activities, 
experience, and characteristics. Interviews provide us an opportunity to 
elaborate attitudes through group and individual discussions'. Online systems 
can possibly be designed to gather attitudinal data from .participants and 
site visitors. The preceding points illustrate the interrelationship of 
instrumentation with areas of inquiry and audience. ^ 



D. Time of Data Acquisition . : 

Another consideration in instrumentation i^the time of administration. 
As mentioned earlier, baseline and concurrent control procedures are to be ten 
integral part of the evaluation. Baselipe measures allow us to establish 
reference points for achievement and attitude before the introduction of 

/ ■ ■ • ■ 
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computQir-assisted instruction to the schools and col^ieges. When computer 
curr^ular materials are in use for student instruct ion » measures obtained 
£r^ students and faculty without direct Involy/inent in the demonstrations 
aj^ovide us with relative standards for concu^enL coq^parisons with results 
'from conq>uter-assi8ted instruction classes y The time selected for instrument 



administration is also crucial t-o the us^^'^of pre- and posttests as indicators 
of student achievement* ^ 

Of course the area of inquiry, ,^udience, and mode of data acquisition 
contribute to the specif icatiqn o]^ 'timing* Surveys of activities are less ' 
sensitive to the "when" dimensiph than achievement tests are. The time 
allotted for a respondent to t^ake a standardized test is less flexible than 
the time allotted to complete an attitude questionnaire^ Requests for 
cooperation must be realJ^tic; we must not intrude upon the audlence^s other ^ 
responsibilities* ' * 



Interrelationships 

^/ > 
Wiile the dimensions of the educational analysis s^re interrelated^ some 

ol the iater^lationships are irrelevant to *the ^evaluation » To convey ^e 

import ol ifhosc interrelationships and to elaborate upon the areas of .inquiry > 

we are supplying the following compreh&nsive outline of potential affective 

indicators, with references to each dimension. This derineat ion. Expands, to 

the level item sterns^ one section of the hierarchical schema/ for areas of 

inquiry. Bie courseware* role» and appraisal of computer-assisted* instruction 

are detailed through successive refinements of their 'respective con^>onents» 

and through the consideration of other dimensions. / 



11 



er|c 



-1. Content. The content of instructional materials refers to (1) the 
jscope cf the subject matter; (2) the eurricular objectives; and (3) the 
preacntation forniat. The scope of the subject matter is the range and depth 
of content covered in instruction. iVidications of that co^a^e include: 
emphaais upon major points; use of examples and illustrations; a^ifeemeuL 
between content objectives and instructional practice; instructional challenge 
of exercises and questions. Students* perceptions of the extent of content 
are influenced by attitudes and anxiety related to the subject area; this 
situation therefcre necessitates the inclusion of items concerning attJ?^ude 
toward the subject matter and anxiety attributed ro th-e difficulty of the 
material. Since content is relevant to both traditional and con^uter- based 
instruction students might respond to questions about the content of tradi- 
tional instruction at . the baseline period. They might respond to questions 
about the content of traditional and computer-based instruction at the 
demonstration period. Another source of data about the content of computer- 
based materials is the participating instructor. 

It is doubtful that Students are competent to evaluate eurricular 
objectives. Here instructors would be the appropriate respondent. For this 
subtopic items include: the clarity of courseware objectives; emphasis upon 
objectives according to a iogicaJ framework; and a statement on the curricalar 
specification required by th«/ courseware. The third subtopic for content 
reliites to the prefsentation of the instructional materials by coj»q>uter. 
Questions to be answered are these: Is the format beyond the capabilities 
of other medlaj^ Are there advantages to computer presentation for this 
content area? Is computer-assisted instruction appropriate for the content? 

12 
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Students and instructors might respond to these presentation inquiries before 
and after the implementation of CAI to indicate attitudinal intact for a- 
conCfUt area. Another important source of data on the content of courseware 
is the subject-matter expert. Courseware review by experts in the content 
areas provides judgmental data to complement response data from students and 
instructors. 

2. Instructional Strategy. The sequence and pacing, feedback, manage- 
ment, and component objectives of instructional materials will depend upon 
the instructional strategy adopted for courseware. Individualization is 
an evasive concept, but sequence and pacing are amenable to an operational 
definition which encompasses the following: adaptation to a particular 
student; imposition of constraints upon learning; resolution of cognitive 
difficulties; progress through famixiar material; encouragement for instruc- 
tional challenge; detection of curricular or component weaknesses; and use of 
theoretical and/or practical criteria. Wich the exception of the last Item 
these points refer to the nature of the interaction between the student and 
the instructional system. Instructors and participating students represent 
appropriate audiences for" respl)Hse roHgwtng^^ urocw a r -e-impJL eme n'ta tlon. Non- 
partir.ipating students might provide information on sequence and pacing for 
conventional instruction during the baseline and demonstration periods. 

Comparisons, between the extent of available feedback. in conventional and 
computer-based settings rely upon student responses. Item-stems relevant 
to feedback are: provision of clarification and guidance; assistance fxom 
comment's regarding answers and status; reinforcement effects; maintenance 

of learner motivation; explanation of objectives;, and, beyond the stu^nt 

» 

audience, the incorporation of diagnostic and administrative Information for 

13 
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instructors. As with other attitudinal inquiries, a questionnaite appears 
to be a suitable form for instrumentation, with the interview reserved for 
analysis in depth. 

The evaluative procedures, instructional manageinent , and convener £ 
objectives inherent to the courseware require review by instructional psychol- 
ogists. These reviews by experts provide judgnjental data to support analytic 
information on achievement and behavior and on validity and reliability. 
For evaluative and management procedures Judgments are expected to cover the 
following: clarity and measurability o£ mastery criteria} agreement with 
course objectives; validity for supervision, placement ... and pacing; conformity 
with theoretical or practical criteria. For objectives concerned with com- 
ponent repertoires points for review include: consistency with instructional 
pstrategy; generality of objectives across content areas; and the theoretical 
or practical basis for objectives. The instrumentation for expert review 
necessitates comprehensive courseware documentation* for a sample segment, or 
a brief demonstration with a concomitant explanation of contingencies, accom- 
panied by an appropriate questionnaire or checklist. The time of review is 
flexible given conq>ieted courseware. 

3. Delivery. Those attitudinal statemenLs deallng~with delivery pertain 
to the mode of contact, facility of operation, and availability. While^facility 
and availability are incorporated in the technical analysis, these subtopics 
of delivery are meant to probe the affective reactions of students. Mode of 
contact entails items- for exploring the personal natu*-e of student-computer, 
exchange » the anxiety attributable both to a technological setting and to an 
authority figure," and student ability to assume motivational responsibility. 
Facility of operation refers to the psychomotor requisites for system operation, 

14 



the visual interpretation of displays, complications introduced by other media, 
and the dramatic effects of the media span upon content. Availability eaconir 
passes: distractions from educational purpose introduced by processing lag 
and by down time, extension of available hours for instruction; accessibility 
of terminals; and the adequacy of space allotment for^. terminals # To descri be 
attitudinal changes toward delivery requires an early and late collection 
of response data during a student *s contact with courseware* 

4. Production. Of the component areas of courseware, production is most 
suitable for instrumentation through interviews with instructors and adminis- 
trators. Among instructors, authors from cooperating schools are an important 
source oi response data on production. The unstructured response format of 
interviews facilitates the expression of feelings and opinions about the 
procedures and process for courseware production. These issues concern the 
appropriate personnel for authoring courseware, means for obtaining professional 
recognition for developing courseware, arid the specifications necessary for 
computer-based curricula. Other ramifications of production would best be 
investigated through questionnaires, particularly questions on the availability 
of instx actional materials and programs, and on the standardization of instruc- 
tional practices. Whether in interviews or questionnaires, items on production 
are scheduled for administratioi during the demonstration period. As might 
be inferred from the above order of consideration, production and delivery are 
subordinate to content and instructional strategy as priorities in the educa- 
tional analysis for courseware. " 

B. Role of CAI 

1. Utilization, Is CAI being properly used in the schools? We intend 
to investigate this question by exploring the extent of, and intact upon, 

IS / 
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student and faculty skepticism. If skepticism within the educational institu- 
tion acts to inhibit the use of CAI, the manifestation and alteration of that 
resistance become important factors for the evaluation of educational effective- 
ness. How computer-based education is to be en^jloyed covers the manner of use, 
commitment of instructional time, users* level of education, content areas, ard 
possible corollaries of utilization.. These subtopics involve respondents,* 
attitudes on the use of CAI. 

Mode refers to the manner of adoption appropriate for this instructional 
technology. The alternatives range from outright rejection to complete adop- , 
tion, a range which suggests the advantages of a summative format for attitude 
assessment. An alUed question is whether CAI is perceived as a revolutionary 
medium or as another instructional resource. Closely related to mode is another 
su'btopic of utilization, time commiti&ent. That is the proportion of student 
time in a class or course to , be spent with CAI. Views on mode and time 
commitment are to be obtained from students, instructors, and administrators 
thTough questionnaires distributed in the baseline and demonstration years. 
Instructors and administrators might also respond to an item concerning the 
use of their time required to become familiar with CAI. 

Undoubtedly, level of education and content area contribute, to determining 
the proper mode arid time commitment for utilization. Through audience classifi- 
cations according to certain characteristics, informati6n on such interactions 
is accumulated without repetitious questioning; responses to each subtopic 
should yield sufficient information. For level of education items concern; 
the stage of student involvement, elementary. school or community college, with 
CAI; the potential of the computer medium for i^nstructing handicapped, employed, 
and distantly located students; the point jm teacher training for acquaintance 

16 
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with CAI. For content area items concern application to a subject area and 
level of that subject suitable for computer-^based instruction. Further 
4iidica4orB^ Include ppsaible corollaries of utilization: reduces requisite 
faculty size; expaiids the capabilities of instructors; allows more flexibility 
and variety in student education; makes students isore active agents in their 
education. The audience and instrumentation for level of education and 
content area follow those for mode and time commitment; corollaries of use 
might be treated' during interviews* 

2» Receptivity. Besides providing baseline data as a control reference, 
receptivity offers an attitudinal reference for utilization tHrou^ a history 
of implementation 9 perception of CAl and conventional education, and concern 
for innovation. In the attitudinal analysis the meaning of^ history is limited 
to the needs, expectations, and responsibilities expressed by participating 
administrators and instructors. Such an attll:udifta3Mv4stA?y^ls to be main- 
tained through various contacts with participants, principally through 
periodic interviews* Items pertinent to perceptions of CAI and conventional 
education are those that explore the following about computer assisted instruc*^ 
tion: it changes role of instructor from purveyor of information to master 
of resources; it releases faculty time for more profitable pursuits; it neglect 
the importance of student-faculty contact for modeling; and it detracts from 
social development by decreasing peer interaction. Although history and 
perception are oriented toward instructors and administrators, inquiries on 
concern for innovation include the student audience. Existing attitude sc'ales 
on concern for innovation are adaptable to our educational analysis. For the 
role of CAI the indicators for utilization take precedence over those for 
receptivity in the allocation of resources. 
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C> Apgraisal of CAI 

1. Priorities. Since audiences hold diffeirent preconceptions and 
perspectives » it appears reasonable to expect variations in priorities for 
their appraisal of effectiveness. Ati audience's priorities should specify 
the importance which it attributes to cost, technology and education. Both 
the individual and relative merits of financial, technical, and educational 
factors in the appraisal of CAI are to be considered by respondents toward 

*' * » 

the conclusion of the demonstrations. Participants' might also indicate - 
their assessments of each factor and their support of CAI» The continuum 
represented by Like rt formats for attitudina*! questionnaires fits the pur- 
poses of inquiries regarding audience priorities. 

2. Basis. Basis for appraisal concerns the previous experience of 
respondents t and their satisfaction and involvement with conventional 
education. Statements regarding previous e^cperience are to be accumulated 
through surveys of activities and behavior > and through school records. 
Although e;Kperience entails matters of fact, subjective responses to tech- 
nical experience might also indicate comfort with instructional technology. 
Attitudinal inquiries in this category include indicators of Involvement 
and satisfaction with conventional instruction. 
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